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I. INTRODUCTION 

In a recent work [l| we proposed quantization rela- 
tionships which would let us describe and solution prob- 
lems originated by conflicting or cooperative behaviors 
among the members of a system from the point of view 
of quantum mechanical interactions. Through these re- 
lationships we could described a system through a den- 
sity operator and its entropy would be given by the von 
Neumann entropy. The quantum version of the replica- 
tor dynamics is the equation of evolution of mixed states 
from quantum statistical mechanics. 

Since Shannon 2], information theory or the mathe- 
matical theory of communication changed from an engi- 
neering discipline that dealed with communication chan- 
nels and codes Q to a physical theory [4] in where the 
introduction of the concepts of entropy and information 
were indispensable to our understanding of the physics of 
measurement. Classical information theory has two pri- 
mary goals : The first is the development of the funda- 
mental theoretical limits on the achievable performance 
when communicating a given information source over 
a given communications channel using coding schemes 
from within a prescribed class. The second goal is the 
development of coding schemes that provide performance 
that is reasonably good in comparison with the optimal 
performance given by the theory. 

Quantum information theory may be defined [6| as the 
study of the achievable limits to information processing 
possible within quantum mechanics. Thus, the field of 
quantum information has two tasks: First, it aims to 
determine limits on the class of information processing 
tasks which are possible in quantum mechanics and pro- 
vide constructive means for achieving information pro- 
cessing tasks. Quantum information theory appears to 
be the basis for a proper understanding of the emerging 
fields of quantum computation [71, |8| , quantum commu- 
nication fil, [KI , and quantum cryptography [ll|, [I4I • En- 
tropy is the central concept of information theories. The 
quantum analogue of entropy appeared 21 years before 
Shannon's entropy and generalizes Boltzmann's expres- 
sion. It was introduced in quantum mechanics by von 
Neumann [13l . Il4| . Entropy in quantum information the- 
ory plays prominent roles in many contexts, e.g., in stud- 




ies of the classical capacity of a quantum channel [1 
and the compressibility of a quantum source llTI . |l 



II. SHANNON, ENTROPY AND CLASSICAL 
INFORMATION THEORY 

Entropy 0, ^M, is the central concept of information 
theory. In classical physics, information processing and 
communication is best described by Shannon information 
theory. 

The Shannon entropy expresses the average informa- 
tion we expect to gain on performing a probabilistic ex- 
periment of a random variable A which takes the value 
fli with the respective probability pi. It also can be seen 
as a measure of uncertainty before we learn the value of 
A. We define the Shannon entropy of a random variable 
A by 



i/(A)=i/(pi,...,p„) 



i=l 



Pi^0g2Pi- 



(1) 



The entropy of a random variable is completely deter- 
mined by the probabilities of the different possible val- 
ues that the random variable takes. Due to the fact that 
p = (pi, ...,p„) is a probability distribution, it must sat- 
isfy Y^^i=iPi — 1 ^^'^ < pi, ...,p„ < 1. The Shannon en- 
tropy of the probability distribution associated with the 
source gives the minimal number of bits that are needed 
in order to store the information produced by a source, in 
the sense that the produced string can later be recovered. 
Shannon formalized the requirements for an information 
measure H{pi, ...,p„) with the following criteria: 

1. H should be continuous in the pi. 

2. If the Pi are all equal, i.e. pi = 1/n, then H should 
be a monotonic increasing function of n. 

3. H should be objective: If a choice be broken down 
into two successive choices, the original H should 
be the weighted sum of the individual values of H. 



H{pi,...,pn) 



H{pi 

+ {Pi- 



'P2,P3, 
P2)H{- 



,Pn) 
Pi 



P2 



Pi +P2 Pl+ P2 



-)• (2) 



Suppose A and B are two random variables. The joint 
entropy H{A, B) measures our total uncertainty about 
the pair (A, B). The joint entropy H{A, B) is defined by 



H{A,B) = -^ p,j log2 pij 



«j 



while 



iJ(A) = -^Pylog2 5^ 



Piji 



*J 



H{.B) = -Y^ p^j l0g2 Y^ Prj , 



(3) 

(4) 
(5) 



«j 



where pij is the joint probability to find A in state a^ and 
B in state bj. 

The conditional entropy H{A | _B) is a measure of how 
uncertain we are about the value of A, given that we 
know the value of B. The entropy of A conditional on 
knowing that B takes the value bj is defined by 



H{A I B) = H{A, B) - H{B), 
HiA\B) = -J2Pv'^og2p. 



«|j' 



(6) 



where piu = ^^^ — is the conditional probability that A 
is in state a^ given that B is in state bj . 

The mutual or correlation entropy H{A : B) oi A and 
B measures how much information A and B have in com- 
mon. The mutual or correlation entropy H{A : B) is 
defined by 



H{A ; B) = H{A) + H{B) - H{A, B), 
H{A : B) = - ^Pjj logaPj: 



"i:j: 



(7) 



where pi;j is the mutual probability defined as pi:j 

J2i P^D E, Pij 



Pa 



The mutual or correlation entropy also can 



be expressed through the conditional entropy via 



H{A : B) 
H{A : B) 



H{A) - H{A I B), 
H{B)-H{B I A). 



(8) 
(9) 



The joint entropy would equal the sum of each of A's 
and -B's entropies only in the case that there are no cor- 
relations between A's and i?'s states. In that case, the 
mutual entropy or information vanishes and we could not 
make any predictions about A just from knowing some- 
thing about B |2fl]. 

The relative entropy H[p \\ q) measures the closeness 
of two probability distributions, p and q, defined over the 
same random variable A. We define the relative entropy 
of p with respect to q by 

H{:P II q) = Ypt\og2Pi -^Pzlogggz, 

i i 

H{p II q) = -H{A) -Y^p, log2 q,. (10) 



The relative entropy is non-negative, H{p \\ q) > 0, with 
equality if and only if p = g. The classical relative en- 
tropy of two probability distributions is related to the 
probability of distinguishing the two distributions after a 
large but finite number of independent samples (Sanov's 
theorem) [3]. 

Lets review some basic properties of entropy [3| : 

1. H{A, B) = H{B, A), H{A : B) = H{B : A). 

2. H{B \ A) >0 and thus H{A : B) < H{B), with 
equality if and only if i? = f(A). 

3. H{A) < H{A, B), with equality if and only if B ^ 

4. Subadditivity: H{A, B) < H{A) + H{B) with 
equality if and only if A and B arc independent 
random variables. 

5. H{B I A) < H{B) and thus H{A : B) > 0, with 
equality in each if and only if A and B are inde- 
pendent random variables. 



6. Strong subadditivity: 
H{A,B)+HiB,C). 



H{A,B,C) + H{B) < 



Conditioning reduces entropy 

H{A\B,C)<H{A\B) (11) 

and for a set of random variables Ai, ...TAn and B, the 
chaining for conditional entropies is 

n 

H{Ai,...,An\B) = Y.H{A, |S,Ai,...,A,_i). (12) 



Suppose A ^ B ^ C is a. Markov chain. Then 

HiA) > H{A : B) > H{A : C), 
H{C : B) > H{C : A). 



(13) 
(14) 

The first inequality (|13[) is saturated if and only if, given 
B, it is possible to reconstruct A. The data process- 
ing inequality (J13p states that the information we have 
available about a source of information can only decrease 
with the time: once information has been lost, it is gone 
forever. If a random variable A is subject to noise, pro- 
ducing B, the data processing cannot be used to increase 
the amount of mutual information between the output 
of the process and the original information A. The data 
pipelining inequality (J14p says that any information C 
shares with A must be information which C also shares 
with B] the information is "pipelined" from A through 
B toC H. 



III. VON NEUMANN, ENTROPY AND 
QUANTUM INFORMATION THEORY 

Von Neumann [l^, 13 defined the entropy of a quan- 
tum state p by the formula 



S{p) = -Triplnp) 



(15) 



which is the quantum analogue of the Shannon entropy 
H ^]. The entropy S{p) is non- negative and takes its 
maximum value Inn when p is maximally mixed, and its 
minimum value zero if p is pure. If Ai are the eigenvalues 
of p then von Neumann's definition can be expressed as 



S{p) 



Y, h In Xi 



(16) 



The von Neumann entropy reduces to a Shannon entropy 
if p is a mixed state composed of orthogonal quantum 
states ^2Qj . If [/ is a unitary transformation, then 

S{p) = S{UpU^). (17) 

If a composite system AB is in a pure state, then S{A) — 
S{B). Suppose p = J2iPiPi where pi are probabilities, 
and pj are density operators. Then 



s{p)<h{p,) + j:p,s{p^) 



(18) 



with equality if and only if the states Pj have support 
on orthogonal subspaces, i.e. suppose \i) are orthogo- 
nal states for a system A, and p^ is any set of density 
operators for another system B. Then 



S{EP^ N) 01 ® P^) = H{p^) + 'ZP^S{P^), 



(19) 



where H(jJi) is the Shannon entropy of the distribution 
Pi. The entropy is a concave function of its inputs. That 
is, given real numbers pi, satisfying pi > 0, J2iPi — ^i 
and its corresponding density operators p,j, the entropy 
satisfies the equation 



S{Y.P^Pi) > llP^S{Pi)- 



(20) 



It means that our uncertainty about this mixture of 
states should be higher than the average uncertainty of 
the states p.^. 

The maximum amount of information that we can ob- 
tain about the identity of a state is called the accessi- 
ble information. It is no greater than the von Neumann 
entro py o f the ensemble's density matrix (Holevo's theo- 
rem) |2ll . |23 . |23| . 12411 and its greatest lower bound is the 
subentropy Q{p) [2^ defined by 



Qip) = - E n 



jAjlnAj. (21) 

A," — Ai 



The upper bound x — S{p)~J2iPi^iPi)^ called Holevo's, 
on the mutual information resulting from the measure- 
ment of any observable, including POVM's, which may 
have more outcomes than the dimensionality of the sys- 
tem being measured is 

HiA:B)<Sip)-j:p,S{pi)<H{A), (22) 

i 

H{A:B)<S{p)<\nn. (23) 



By analogy with the Shannon entropies it is possible 
to define conditional, mutual and relative entropies 

S{A I B) = S{A, B) - S{B), (24) 

S{A:B)=S{A) + S{B)^S{A,B), (25) 

S{A:B) = S{A)-S{A\B), (26) 

S{A:B) = S{B)-S{B\A). (27) 

The negativity of the conditional entropy always indi- 
cates that two systems are entangled and indeed, how 
negative the conditional entropy is provides a lower 
bound on how entangled the two systems are Q . 

The von Neumann entropy is additive, it means that 
if p^ is a state of system A and p^ a state of system B, 
then 



S{A®B) ^S(A) + S{B) 



(28) 



and strongly subadditive, which means that for a tripar- 
tite system in the state Pabc 



S{A, B, C) + S{B) < S{A, B) + S{B, C). 



(29) 



Suppose distinct quantum systems A and B have a joint 
state Pab- The joint entropy for the two systems satisfies 
the next inequalities 



S{A,B) <S{A) + S{B), 
SiA,B) > \SiA)-SiB)\. 



(30) 
(31) 



The first inequality is known as subadditivity, and it 
means that there can be more predictability in the whole 
than in the sum of the parts. The second inequality is 
known as triangle inequality. 

Let p and a be density operators. We define the rela- 
tive entropy [26| of p with respect to a to be 

5'(p|| cr) =rr(plnp)-rr(plncr). (32) 

This function has a number of useful properties [l9| : 

1. S{p II a) > 0, with equality if and only if p = cr. 

2. S{p II (t) < oo, if and only if suppp C suppa. (Here 
"supp" is the subspace spanned by eigenvectors of 
p with non-zero eigenvalues). 

3. The relative entropy is continuous where it is not 
infinite. 

4. The relative entropy is jointly convex in its argu- 
ments [23|. That is, if pj^ P2;Cri and (72 are density 
operators, and pi and p2 are non-negative numbers 
that sum to unity (i.e., probabilities), then 



Sip II cr) < Pi5'(pi II o-i) +P25'(p2 II 0-2) 



(33) 



where p = PiPi + P2P2 Eind a — picri +P20'2- Joint 
convexity automatically implies convexity in each 
argument, so that 



S{p II (j) < PiS{pi II cr) +p2S{p2 II cr). 



(34) 



Sanov's theorem Q has its quantum analogue [2^, [23| ■ 
Suppose p and a are two possible states of the quantum 
system Q, and suppose we are provided with N identi- 
cally prepared copies of Q. A measurement is made to 
determine whether the prepared state is p. The proba- 
bility Pn that the state a is confused with p is 



P, 



N 



-NS(p\W) 



(35) 



The relative entropy can be seen as a measure of "dis- 
tance" or of separation between of two density operators. 
Two states are "close" if they are difficult to distinguish, 
but "far apart" if the probability of confusing them is 
small [26] , 
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